Integration of the 3D Environment for UAV Onboard Visual Object Tracking

Single visual object tracking from an unmanned aerial vehicle (UAV) poses fundamental challenges such as object occlusion, small-scale objects, background clutter, and abrupt camera motion. To tackle these difficulties, we propose to integrate the 3D structure of the observed scene into a detection-by-tracking algorithm. We introduce a pipeline that combines a model-free visual object tracker, a sparse 3D reconstruction, and a state estimator. The 3D reconstruction of the scene is computed with an image-based Structure-from-Motion (SfM) component that enables us to leverage a state estimator in the corresponding 3D scene during tracking. By representing the position of the target in 3D space rather than in image space, we stabilize the tracking during ego-motion and improve the handling of occlusions, background clutter, and small-scale objects. We evaluated our approach on prototypical image sequences, captured from a UAV with low-altitude oblique views. For this purpose, we adapted an existing dataset for visual object tracking and reconstructed the observed scene in 3D. The experimental results demonstrate that the proposed approach outperforms methods using plain visual cues as well as approaches leveraging image-space-based state estimations. We believe that our approach can be beneficial for traffic monitoring, video surveillance, and navigation.Comment: Accepted in MDPI Journal of Applied Science

READMem: Robust Embedding Association for a Diverse Memory in Unconstrained Video Object Segmentation

We present READMem (Robust Embedding Association for a Diverse Memory), a modular framework for semi-automatic video object segmentation (sVOS) methods designed to handle unconstrained videos. Contemporary sVOS works typically aggregate video frames in an ever-expanding memory, demanding high hardware resources for long-term applications. To mitigate memory requirements and prevent near object duplicates (caused by information of adjacent frames), previous methods introduce a hyper-parameter that controls the frequency of frames eligible to be stored. This parameter has to be adjusted according to concrete video properties (such as rapidity of appearance changes and video length) and does not generalize well. Instead, we integrate the embedding of a new frame into the memory only if it increases the diversity of the memory content. Furthermore, we propose a robust association of the embeddings stored in the memory with query embeddings during the update process. Our approach avoids the accumulation of redundant data, allowing us in return, to restrict the memory size and prevent extreme memory demands in long videos. We extend popular sVOS baselines with READMem, which previously showed limited performance on long videos. Our approach achieves competitive results on the Long-time Video dataset (LV1) while not hindering performance on short sequences. Our code is publicly available.Comment: Accepted to BMVC 2023. Code @ https://github.com/Vujas-Eteph/READMe

Estimating the precision of under-water video-mosaics

Under-water video-mosaics are an important tool e.g. for inspection of man-made infrastructure. Cameras may drift in rotation and distance to the surface while the mosaic will often be very much larger than a single frame resulting in long chains of planar homographies. This contribution addresses the problems arising from dead-reckoning drift in such chains. Local patches are rectified using homography decomposition. Experiments with a portal allowing underwater image sequences with mechanical groundtruth are performed. Thus the deviation of the mosaic from a true orthographic map may be estimated

Designing observer trials for image fusion experiments with Latin Squares

Multisensor image fusion (e.g. IR with visual) is the process of combining relevant information from two or more images into a single image. The aim is to find an objective quality measure, which can be used in automatic applications, that correlates best with subjective observer trials. Not all combinations of image, algorithm, and test observer can be worked out. In this paper R. Fisher’s Design of Experiments approach based on Latin Squares is used for thinning out the number of experiments for each observer in such observer trials while preserving exactness and reliability of the result

Simulation of the interaction of a high energy laser beam with the sea surface in the short wavelength infrared

The knowledge of the interaction of a high energy laser beam with a dynamic sea surface is of great practical interest in maritime environments. The components transmitted into the sea and reflected at the sea surface have to be considered. The calculation of energy transfer into the sea is fundamental to the prediction of upper-ocean heating and temperature-dependent optical properties of the sea, which in turn influence its reflectance characteristics. In addition, the spatial energy (or power) distribution of the laser beam reflected at the dynamic sea surface is also of high significance. For the estimation of the laser light energy reflected into a specific spatial direction, several parameters need to be considered, e.g., wind speed, wind direction, and fetch. The calculated amount of light energy reflected into a specific direction varies statistically and depends largely on the dynamics of the wavy sea surface. A 3D simulation of a dynamic sea surface is presented interacting with a high energy laser beam in the short wavelength infrared spectral band. The simulation computes the upper-ocean heating, the temperature-dependent Fresnel reflectances, and the absorption in seawater considering the laser geometric configuration. For the reflectance calculations, a bistatic configuration of the laser source and receiver is regarded, where the receiver positions are on a virtual hemisphere having the laser spot center as the center point. The specular reflection of the laser beam at the sea surface is modeled by an analytical statistical bidirectional reflectance distribution function of the sea surface. The simulation is restricted to sea surfaces heated to the boiling point to avoid complex phase transition effects between water and gas. For a high energy laser beam focused on a small laser spot on the evolving wavy sea surface, the maximum expected reflected laser power is calculated for the specular forward- and back-reflection direction for glints. The probability of occurrence and temporal occurrence of those glint events is estimated for both directions

Comparing visual tracker fusion on thermal image sequences

Visual object tracking is a challenging task in computer vision, especially if there are no constraints to the scenario and the objects are arbitrary. The number of tracking algorithms is very large and all have diverse advantages and disadvantages. Normally they show various behaviour and their failures in the tracking process occur at different moments in the sequence. So far, there is no tracker which can solve all scenarios robustly and accurately. One possible approach to this problem is using a whole collection of tracking algorithms and fusing them. There exist various strategies to fuse tracking algorithms. In some of them only the resulting outputs are fused. This means that new algorithms can be integrated with less effort. This fusion can be called "high-level" because the tracking algorithms only interact through the last step in their procedure. Three fusion methods are investigated. They are called Weighted mean fusion, MAD fusion and attraction field fusion. In order to evaluate the three different approaches a collection of thermal image sequences has been investigated. These sequences show maritime scenarios with various objects such as ships and other vessels